Multiple Visual Representations Research Articles

Continuous lipreading based on acoustic temporal alignments

Visual speech recognition (VSR) is a challenging task that has received increasing interest during the last few decades. Current state of the art employs powerful end-to-end architectures based on deep learning which depend on large amounts of data and high computational resources for their estimation. We address the task of VSR for data scarcity scenarios with limited computational resources by using traditional approaches based on hidden Markov models. We present a novel learning strategy that employs information obtained from previous acoustic temporal alignments to improve the visual system performance. Furthermore, we studied multiple visual speech representations and how image resolution or frame rate affect its performance. All these experiments were conducted on the limited data VLRF corpus, a database which offers an audio-visual support to address continuous speech recognition in Spanish. The results show that our approach significantly outperforms the best results achieved on the task to date.

Investigating students’ meaning-making of multiple visual representations of epigenetics at different levels of biological organisation

ABSTRACT The aim of this study is to investigate students’ meaning–making of multiple visual representations of epigenetics at different levels of biological organisation, and to discern what visual aspects of the multiple visual representations might influence students’ reasoning. Adopting an exploratory approach, we analysed how students made meaning of visually communicated epigenetics phenomena while pointing at and reasoning about the multiple visual representations as part of semi-structured focus groups. We investigated students’ meaning-making of the multiple visual representations by analysing their indications through physical pointing and accompanying verbal utterances. The analysis revealed meaning-making and the nature of linking between levels of organisation in four distinct patterns, namely intra horizontal linking, inter horizontal linking, one level vertical linking and two level vertical linking. In addition, five different visual characteristics of the multiple visual representations emerged as influencing students’ reasoning while linking between different organisation levels: multiple visual representations, salient visual features, analogous visual features, familiar visual elements, and textual adjuncts. The study shows that multiple visual representations at different levels of organisation can support students’ meaning-making of epigenetics, indicating that this way of communicating can be transferable to other biological domains. Potential implications for future research and teaching practice are provided.

Alternating capture of attention by multiple visual working memory representations

Can we look for multiple objects simultaneously? Previous studies have proposed that the representation of an item in visual working memory (VWM) can bias the deployment of attention to memory-matching items in visual search. However, it is still controversial whether multiple VWM item representations are able to capture attention. In the present study, we adopted an eye-tracking technique to reveal this issue. In Experiment 1, we replicated Chen and Du’s Experiment 2 and adopted an eye-tracking technique to determine whether multiple VWM item representations are able to bias attention. In Chen and Du’s paradigm, the memory test was always followed by the search test, and the participants might intentionally prepare for the search task, which can affect the results of the research. Thus, in Experiment 2, we prevented participants from predicting the temporal occurrence of the visual search task by randomly conducting a search test or a memory test after memoranda. The findings of the present study suggested that only one working memory item at a time influences attention and multiple working memory items may affect attention through alternation.Protocol registrationThe stage 1 protocol for this Registered Report was accepted in principle on September 5, 2022. The protocol, as accepted by the journal, can be found at: https://doi.org/10.17605/OSF.IO/BE529.

Efficient Shot Boundary Detection with Multiple Visual Representations

Due to the unlimited growth of video-capturing devices and media, searching and finding a particular video in this huge database becomes a laborious as well as expensive task. Information-rich shots are the inevitable factor of the content-based video processing (CBVP) system. Hence, shot boundary detection (SBD) becomes the basic step of all content-based video retrieval processes. The accuracy of the existing SBD methods highly suffers from false positives and false negatives due to the presence of multiple variants. An efficient SBD method with multiple invariant features is proposed in this paper. A right combination of invariant features such as edge change ratio (ECR), colour layout descriptor (CLD), and scale-invariant feature transform (SIFT) key point descriptors helped to improve the accuracy level of SBD. As the selected features are invariant to most of the variants in video frames, such as illuminance changes, motion, scaling, and rotation, a markable reduction in false detection is possible. Support vector machine (SVM) classifier is used for the classification of frames into transition frames and shot frames. This proposed method is experimented and analysed with the standard SBD dataset TRECVid 2007 videos. The experimental results are compared with some state-of-art methods, and our method shows better performance with a 97% of F1 score.

Teaching Whole Number Addition and Subtraction to Students With Learning Disabilities

Students with learning disabilities in mathematics often struggle with the underlying concepts of multidigit addition and subtraction. To help students build a conceptual understanding of these computations, teachers can utilize evidence-based practices such as the concrete–semi-concrete–abstract framework and the use of multiple visual representations. This column presents five key strategies that incorporate evidence-based practices and teach whole number operations that rely on place value understanding.

Dipole Radiation – Multiple Visual Representations to Assist Learning

The Hertzian Dipole is a standard topic for lectures in physics. It points to the fundamental mechanism that is responsible for electromagnetic radiation. Understanding the Hertzian Dipole is the first step toward understanding toward understanding the physics behind different antenna types, such as half-wave antennas and other types. But it is not easy to discuss the spatial and time-dependent behaviour.Suitable illustrations and animations showing the formation of electromagnetic waves can help for teaching and learning. Moreover, a combination of multiple graphical representations can bring to light the relationships between different aspects. An HTML5-applet was written, specifically to illustrate dipole radiation. The program “DipoleRadiation” is designed to make aspects of electromagnetic radiation visible that cannot be seen directly.In an explorative pilot study students were interviewed to find out more about their problems. Furthermore, an eye-tracker study was carried out to identify, what learners focussed on. First findings revealed some gaps in knowledge. Among them are a simple lack of prior knowledge, misunderstanding of models, invisible features, spatial imagination, and the ability to identify important details and abstractions from the superficial.

Interactive Data and Information Visualization: Unpacking its Characteristics and Influencing Aspects on Decision-making

Background: Interactive data and information visualization (IDIV) enhances information presentations by providing users with multiple visual representations, active controls, and analytics. Users have greater control over IDIV presentations than standard presentations and as such IDIV becomes a more popular and relevant means of supporting data analytics (DA), as well as augmenting human intellect. Thus, IDIV enables provision of information in a format better suited to users’ decision-making.

Women and Work in Premodern Europe. Experiences, Relationships and Cultural Representation, c. 1100–1800 ed. by Merridee L. Bailey, Tania M. Colwell, and Julie Hotchin

Reviewed by: Women and Work in Premodern Europe. Experiences, Relationships and Cultural Representation, c. 1100–1800 ed. by Merridee L. Bailey, Tania M. Colwell, and Julie Hotchin Susan Broomhall Bailey, Merridee L., Tania M. Colwell, and Julie Hotchin, eds, Women and Work in Premodern Europe. Experiences, Relationships and Cultural Representation, c. 1100–1800 Abingdon, Routledge, 2018; hardback; pp. 244; 10 b/w illustrations; R.R.P. £115.00; ISBN 9781138202023. As the title suggests, this is a work with wide-ranging ambitions about its contribution to the historiography of women and work. Seeking to take scholarly analysis of women's labouring activities beyond the economic, the essays in this volume variously consider how women negotiated the parameters shaping their intellectual, cultural, emotional, and economic labour. The introduction insists upon the chapters' collective encouragement to us to rethink, refine, and reshape previous scholarly assumptions and approaches to aspects of women's work. As a whole, the collection produces no new meta-narratives, but instead burrows into the complexity of specific contexts, differing individual experiences, varied intellectual constructions, and multiple visual representations, 'giving full credibility to the diversity of premodern women's experiences of work' (p. 21). Its coverage is expansive in time and geography but also in approach. Some chapters examine representations of women's writing as forms of labour and others how fictive women workers were voiced in literary texts. We learn more about how bourgeois wives were recognized as contributing to domestic and economic work, in their household activities for the domestic economy and as outsourced labourers of textile production (an industry that employed men as well as women). Further studies explore women's participation in monastic governance and their roles in guild and civic work. Others consider those making ends meet by begging and vagabondage as forms of agency whether in authorized municipal roles or doing so illicitly. A number of themes run across the volume—lived realities as well as their representations in archives, song, literature and image; and work relationships, dependencies, and networks between women, and women and men, in households and occupational groupings, as patrons and clients, and as intermediaries. Overall, women's work in textiles is particularly foregrounded, as are women of the pen producing literature and letters. In these areas, there is some capacity to see commonalities and differences across the essays, but the ten chapters of this collection are not structured in sub-sections to orient readers towards particular comparative analyses. Its goal is rather to deepen the field with new information and analyses. Yet, although its orientation is towards unsettling present historiography and approaches, and to expand, deepen, and complicate conceptualizations of premodern women's work, I would still have welcomed more from the editors about the global contribution of their new interventions. Do they see changes and continuities across the period the volume covers, between the geographies, [End Page 252] including town and country, or as a result of the religious changes that occurred in the period? In the end, it is thus the details within individual papers that contribute most powerfully, with each study providing important findings and the impetus to explore new directions. Susan Broomhall The University of Western Australia Copyright © 2019 Susan Broomhall

Participatory mapping and food‐centred justice in informal settlements in Nairobi, Kenya

Food vendors are pivotal in the local food system of most low‐income informal settlements in Nairobi, Kenya, despite being seen as an obstruction and as agents of disease and filth by city authorities. This paper explores the geography of these foodscapes – defined as public sites of food production and consumption – in selected low‐income settlements in Nairobi, focusing on the interaction of food vendors with their surrounding environment and infrastructure services. The research uses participatory geographic information system tools, including food mapping with mobile apps and high‐resolution community aerial views with balloon mapping, to capture and contextualise local knowledge. The community mappers collected data on 660 vendors from 18 villages in Kibera, Mathare, and Mukuru, and situated them on multi‐layered synoptic geographic overviews for each settlement. The resulting data on hazardous areas in relation to food spaces and infrastructure provision allowed local communities to prioritise areas for regular clean‐up activities and assisted advocacy to improve these places in cooperation with local authorities. These multiple visual representations of foodscapes make local food vendors, and the risks they face, visible for the first time. Reframing their “right to safe food and environment” from a social and environmental justice perspective allows local communities to put their experiences, knowledge, and challenges faced at the forefront of urban development planning, policy, and practice.

Combining Instructional Activities for Sense-Making Processes and Perceptual-Induction Processes Involved in Connection-Making Among Multiple Visual Representations

Connection-making among multiple representations is a crucial but difficult competence in STEM learning. Prior research has focused on one type of learning process involved in connection-making: sense-making processes leading to conceptual understanding of connections. Yet, other research suggests that a second type of learning process is important: inductive learning processes leading to perceptual intuitions about connections. We investigate whether combining instructional activities designed to support sense-making processes for understanding of connections (understanding activities) and instructional activities that support inductive processes for perceptual intuitions about connections (perception activities) enhances students’ learning of chemistry knowledge. A laboratory-based experiment with 117 undergraduate students compared students in (a) a control condition that received only conventional activities that did not require connection-making; (b) a condition that received conventional and understanding-activities; (c) a condition that received conventional and perception-activities; and (d) a combined condition that received conventional, understanding-activities, and perception-activities. Results show that only the combined condition outperformed the control condition on a test of chemistry knowledge. Eye-gaze data and verbal reports show that understanding-activities and perception-activities have complementary effects on how students integrate information from multiple representations during the learning phase. Finally, we found that students’ spatial skills moderate their benefit from understanding-activities and perception-activities.

Sequencing support for sense making and perceptual induction of connections among multiple visual representations.

Making connections among multiple visual representations is key to students’ learning. This article considers two learning processes involved in connection making: explicit sense making of connections and implicit perceptual induction of connections. Instructional interventions support these processes via different problem types: sense-making problems ask students to verbally explain connections, whereas perceptual-induction problems ask students to quickly categorize numerous representations. Prior research shows that sense-making and perceptual-induction problems enhance students’ learning of domain knowledge—but only if both problem types are combined. Thus, one must ask whether the ability to make sense of connections enhances students’ ability to perceptually induce connections, or vice versa, and, consequently, which problem type students should receive first. This article investigates these questions in the context of undergraduate chemistry. Three experiments with 691 students compare instructional sequences that provide sense-making problems first or perceptual-induction problems first. Effects are assessed on problem-solving performance during the intervention and on chemistry knowledge learning gains after the intervention. Causal path analyses test whether working on sense-making problems first enhances (or impedes) students’ learning from subsequent perceptual-induction problems and vice versa. Results show that sequence interacts with prior chemistry knowledge: High prior-knowledge students show higher learning gains if they receive sense-making problems first. Low-prior-knowledge students show higher learning gains if they receive perceptual-induction problems first. Causal path analyses suggest that sense-making problems enhance students’ learning from subsequent perceptual-induction problems more so than the other way around. However, costs of switching between problem types interfere with low-prior-knowledge students’ ability to take advantage of this effect. (PsycINFO Database Record (c) 2018 APA, all rights reserved)

Understanding Image Impressiveness Inspired by Instantaneous Human Perceptual Cues

With the explosion of visual information nowadays, millions of digital images are available to the users. How to efficiently explore a large set of images and retrieve useful information thus becomes extremely important. Unfortunately only some of the images can impress the user at first glance. Others that make little sense in human perception are often discarded, while still costing valuable time and space. Therefore, it is significant to identify these two kinds of images for relieving the load of online repositories and accelerating information retrieval process. However, most of the existing image properties, e.g., memorability and popularity, are based on repeated human interactions, which limit the research and application of evaluating image quality in terms of instantaneous impression. In this paper, we propose a novel image property, called impressiveness, that measures how images impress people with a short-term contact. This is based on an impression-driven model inspired by a number of important human perceptual cues. To achieve this, we first collect three datasets in various domains, which are labeled according to the instantaneous sensation of the annotators. Then we investigate the impressiveness property via six established human perceptual cues as well as the corresponding features from pixel to semantic levels. Sequentially, we verify the consistency of the impressiveness which can be quantitatively measured by multiple visual representations, and evaluate their latent relationships. Finally, we apply the proposed impressiveness property to rank the images for an efficient image recommendation system.

Supplemental Material for Sequencing Support for Sense Making and Perceptual Induction of Connections Among Multiple Visual Representations

Supplemental Material for Sequencing Support for Sense Making and Perceptual Induction of Connections Among Multiple Visual Representations

Making connections among multiple visual representations: how do sense-making skills and perceptual fluency relate to learning of chemistry knowledge?

To learn content knowledge in science, technology, engineering, and math domains, students need to make connections among visual representations. This article considers two kinds of connection-making skills: (1) sense-making skills that allow students to verbally explain mappings among representations and (2) perceptual fluency in connection making that allows students to fast and effortlessly use perceptual features to make connections among representations. These different connection-making skills are acquired via different types of learning processes. Therefore, they require different types of instructional support: sense-making activities and fluency-building activities. Because separate lines of research have focused either on sense-making skills or on perceptual fluency, we know little about how these connection-making skills interact when students learn domain knowledge. This article describes two experiments that address this question in the context of undergraduate chemistry learning. In Experiment 1, 95 students were randomly assigned to four conditions that varied whether or not students received sense-making activities and fluency-building activities. In Experiment 2, 101 students were randomly assigned to five conditions that varied whether or not and in which sequence students received sense-making and fluency-building activities. Results show advantages for sense-making and fluency-building activities compared to the control condition only for students with high prior chemistry knowledge. These findings provide new insights into potential boundary conditions for the effectiveness of different types of instructional activities that support students in making connections among multiple visual representations.

An adaptive collaboration script for learning with multiple visual representations in chemistry

Undergraduate STEM instruction increasingly uses educational technologies to support problem-solving activities. Educational technologies offer two key features that may make them particularly effective. First, most problem-solving activities involve multiple visual representations, and many students have difficulties in understanding, constructing, and connecting these representations. Educational technologies can provide adaptive support that helps students make sense of visual representations. Second, many problems with visual representations involve collaboration. However, students often do not collaborate effectively. Educational technologies can provide collaboration scripts that adaptively react to student actions to prompt them to engage in specific effective collaborative behaviors. These observations lead to the hypothesis we tested: that an adaptive collaboration script enhances students' learning of content knowledge from visual representations. We conducted a quasi-experiment with 61 undergraduate students in an introductory chemistry course. A control condition worked on a traditional worksheet that asked students to collaboratively make sense of connections among multiple visual representations. An experimental condition worked on the same problems embedded in an educational technology that provided an adaptive collaboration script. The experimental condition showed significantly higher learning gains on a transfer test immediately after the intervention and on complex concepts on a midterm exam three weeks later.

Do Knowledge-Component Models Need to Incorporate Representational Competencies?

Traditional knowledge-component models describe students’ content knowledge (e.g., their ability to carry out problem-solving procedures or their ability to reason about a concept). In many STEM domains, instruction uses multiple visual representations such as graphs, figures, and diagrams. The use of visual representations implies a “representation dilemma”: students learn new content from visual representations they may not yet understand at the same time as they learn about visual representations that show content they do not yet understand. Therefore, students’ learning of content knowledge and of representational competencies (i.e., knowledge about representations) is invariably intertwined. Consequently, instruction may need to adapt not only to students’ acquisition of content knowledge but also to their acquisition of representational competencies. This claim corresponds to the hypothesis that knowledge-component models that describe content knowledge and representational competencies should be more accurate than knowledge-component models that describe only content knowledge. Yet, this hypothesis has not yet been tested. The work in this article tests this hypothesis by comparing knowledge-component models that describe representational competencies and content knowledge to knowledgecomponent models that describe only content knowledge. Analysis of log data from two experiments on chemistry learning with overall 203 undergraduate students suggests that including representational competencies into knowledge-component models increases model fit if the representational competencies are difficult. This finding suggests that students can learn abstract content knowledge only if they have a prerequisite level of representational competencies, and that educational technologies should use adaptive knowledge-component models that capture representational competencies the student has not yet mastered.

Multiple Visual Field Representations in the Visual Wulst of a Laterally Eyed Bird, the Zebra Finch (Taeniopygia guttata)

The visual wulst is the telencephalic target of the avian thalamofugal visual system. It contains several retinotopically organised representations of the contralateral visual field. We used optical imaging of intrinsic signals, electrophysiological recordings, and retrograde tracing with two fluorescent tracers to evaluate properties of these representations in the zebra finch, a songbird with laterally placed eyes. Our experiments revealed that there is some variability of the neuronal maps between individuals and also concerning the number of detectable maps. It was nonetheless possible to identify three different maps, a posterolateral, a posteromedial, and an anterior one, which were quite constant in their relation to each other. The posterolateral map was in contrast to the two others constantly visible in each successful experiment. The topography of the two other maps was mirrored against that map. Electrophysiological recordings in the anterior and the posterolateral map revealed that all units responded to flashes and to moving bars. Mean directional preferences as well as latencies were different between neurons of the two maps. Tracing experiments confirmed previous reports on the thalamo-wulst connections and showed that the anterior and the posterolateral map receive projections from separate clusters within the thalamic nuclei. Maps are connected to each other by wulst intrinsic projections. Our experiments confirm that the avian visual wulst contains several separate retinotopic maps with both different physiological properties and different thalamo-wulst afferents. This confirms that the functional organization of the visual wulst is very similar to its mammalian equivalent, the visual cortex.

Conditions for the Effectiveness of Multiple Visual Representations in Enhancing STEM Learning

Visual representations play a critical role in enhancing science, technology, engineering, and mathematics (STEM) learning. Educational psychology research shows that adding visual representations to text can enhance students’ learning of content knowledge, compared to text-only. But should students learn with a single type of visual representation or with multiple different types of visual representations? This article addresses this question from the perspective of the representation dilemma, namely that students often learn content they do not yet understand from representations they do not yet understand. To benefit from visual representations, students therefore need representational competencies, that is, knowledge about how visual representations depict information about the content. This article reviews literature on representational competencies involved in students’ learning of content knowledge. Building on this review, this article analyzes how the number of visual representations affects the role these representational competencies play during students’ learning of content knowledge. To this end, the article compares two common scenarios: text plus a single type of visual representations (T+SV) and text plus multiple types of visual representations (T+MV). The comparison yields seven hypotheses that describe under which conditions T+MV scenarios are more effective than T+SV scenarios. Finally, the article reviews empirical evidence for each hypothesis and discusses open questions about the representation dilemma.

Managing Spatial Selections With Contextual Snapshots.

Spatial selections are a ubiquitous concept in visualization. By localizing particular features, they can be analysed and compared in different views. However, the semantics of such selections often depend on specific parameter settings and it can be difficult to reconstruct them without additional information. In this paper, we present the concept of contextual snapshots as an effective means for managing spatial selections in visualized data. The selections are automatically associated with the context in which they have been created. Contextual snapshots can also be used as the basis for interactive integrated and linked views, which enable in-place investigation and comparison of multiple visual representations of data. Our approach is implemented as a flexible toolkit with well-defined interfaces for integration into existing systems. We demonstrate the power and generality of our techniques by applying them to several distinct scenarios such as the visualization of simulation data, the analysis of historical documents and the display of anatomical data.

A model of how working memory capacity influences insight problem solving in situations with multiple visual representations: An eye tracking analysis

Insight problem solving, which involves the restructuring of problems and insights, should be closely related to attention and working memory (WM). This study aimed to employ eye-tracking techniques to understand the process by which attention and WM capacity may influence insight problem solving when situations with multiple visual representations are employed. Fourteen graduate students participated in a 70-minute experimental session in this study. The adapted situation-based creativity task (SCT) and the adapted situation-based WM task (SWMT) were employed to measure WM capacity and insight problem solving. Using situation-based visual WM tasks and insight problem solving the findings of this study suggest the following. First, fixation, gaze duration, and saccades to targets are effective eye movement indicators that can aid in the understanding of the cognitive processes of WM and insight problem solving. Second, attention, eye movements, and WM capacity interactively influence insight problem solving, and that influence varies with WM capacity and the insight stage. Accordingly, we propose three stages of insight processes based on eye movements.